Evict a full store down to a low-water mark, and keep draining across restarts - #14

Problem

Once the workerd collector's store reaches its size bound, every export pays for an eviction pass, forever. Executor self-host bundles this collector with a 1 GiB bound. Users report an idle container at about 13% of a core, with 130–200 GiB a day of writes, once the store is full (UsefulSoftwareCo/executor#2194).

Cause

• MotelCollector.write runs holdSizeBound after every export. Once usedBytes is over the bound, that runs cleanupExpired.
• A pass converts only the excess bytes into spans to evict. For one small export, that is the oldest one or two traces. The pass stops as soon as the store is back at the bound.
• Deleting a couple of traces rarely frees whole pages. Their index entries are spread across the attribute and summary indexes, so usedBytes barely moves and the store stays at or just over the bound. The next export crosses it again.
• Each pass also costs time in proportion to the whole store, not to what it evicts. DELETE FROM span_operation_fts WHERE trace_id IN (…) reads the entire operation index, because trace_id is an UNINDEXED FTS5 column. On top of that come SUM(span_count) over trace_summaries, the orphan scans, and repeated iterations within the 500 ms budget.

The result is one full-store pass per export, which is about one a second on an idle Executor.

Fix

This builds on #11, rebased onto main and kept as its own commit with one fix on top.

• Evict down to a low-water mark instead of once per export #11's commit: a store over a bound evicts down to a low-water mark: 90% of the bound by default, set by MOTEL_OTEL_EVICTION_LOW_WATER_PERCENT. After that, exports find room until the store fills again. /api/ingest reports evictions (passes, traces).
• Fix on top: the "draining" state is now stored in motel_maintenance. At a 1 GiB bound, a drain takes several 500 ms passes. workerd can unload the collector between them; with an in-memory flag, the drain then stopped at about 1011 MiB instead of 922.

The bound itself still holds after every export, so holdSizeBound stays on the ingest path.

Supersedes #11 and #12. Both are based on 7cbec5a, which predates #13.

Before / after

Reproduced with the built workerd collector, the bundled 1 GiB bound and product-shaped spans. The store held 264–269k spans; the reporter's store held 268k. After filling past the bound, an idle stream sent one trace a second. Numbers are the workerd process's /proc deltas over a 90–120 s steady-state window, after any drain.

The same stream on an almost empty store: 0.7% CPU, 1.8 MiB/s of writes at 17 spans/s; 0.3% CPU, 0.38 MiB/s of writes at 2 spans/s.

Not in the "after" rows: each drain from the bound to 90% evicts about 2,200 traces in about 8 passes. At 17 spans/s that wrote about 1.2 GiB over two minutes, and the store needs about 25 minutes to refill. Averaged over a refill, that adds about 0.8 MiB/s at 17 spans/s and about 0.1 MiB/s at 2 spans/s.

Writes fall less than CPU and reads. That is expected: in steady state the store must delete as much as it ingests, so those writes scale with the span rate, not with how evictions are batched.

The before 2 spans/s windows overlapped other test runs on the host, so its CPU and p95 figures are noisy.

Not changed here (follow-ups)

• Per-export write cost. Ingest itself writes about 300 KiB per export, even for a 2-span export on an empty store:

  • about 32 WAL pages;
  • a sub-journal of about 146 KiB in a fresh /var/tmp/etilqs_* file per export (transactionSync inside workerd's implicit transaction).

  It also touches many random-key indexes. idx_span_attributes_trace_span duplicates the (trace_id, span_id, key) primary key's prefix, and is 148 MiB of a full 1 GiB store.

• Pass cost still O(store). The operation index delete during a drain, and the orphan log scans every alarm, still read the whole store. Fixing the first needs the FTS rowid tied to spans.rowid, which means a one-off index rebuild.

• Broken FTS merge. incrementalFtsMerge sends INSERT INTO t(t) VALUES('merge=100'), which SQLite rejects ("SQL logic error"), and the error is swallowed. The intended command is INSERT INTO t(t, rank) VALUES('merge', 100).

Verified: tsc --noEmit; bun test: 93 pass across the Bun-side suites, 7 pass across workerd.bounds and workerd.integration, including the test in the PR above.